Developing multimodal intelligent affective interfaces for tele-home health care

Accounting for a patient’s emotional state is integral in medical care. Telehealth research attests to the challenge clinicians must overcome in assessing patient emotional state when modalities are limited (Pettinari and Jessopp, 2001). The extra effort involved in addressing this challenge requires attention, skill, and time. Large caseloads may not afford tele-home health care clinicians the time and focus necessary to accurately assess emotional states and trends. Unstructured interviews with experienced tele-home health care providers support the introduction of objective indicators of patients’ emotional status in a useful form to enhance patient care. We discuss our contribution to addressing this challenge, which involves building user models not only of the physical characteristics of users – in our case patients – but also models of their emotions. We explain our research in progress on Affective Computing for tele-home health-care applications, which includes: developing a system architecture for monitoring and responding to human multimodal affect and emotions via multimedia and empathetic avatars; mapping of physiological signals to emotions and synthesizing the patient’s affective information for the health-care provider. Our results using a wireless non-invasive wearable computer to collect physiological signals and mapping these to emotional states show the feasibility of our approach, for which we lastly discuss the future research issues that we have identified.